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ABSTRACT • 

A reorganized version- of standards to be utilized for 
those who are involved with educational evaluation in Brazil was the 
result of a critical review of evaluation standards which were 
originally elaborated, by the Joint Committee on Standards for 
Educational Evaluation .'The conceptual framework for the critical 
review comprised logical principles. Different methodological 
procedures — connection, complementation and formulation — were applied 
during this review. The reorganized version includes 20 standards: 
six related to utility aspects, three concerned with feasibility, 
five dealing with propriety features, and six related to accuracy 
characteristics of evaluations. (Author) 
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Evaluation Standards: 
Reorganization for the 
Brazilian Educational Reality 

Terezinha Rodrigues de Oliveira 
Ligia Gomes Elliot 
Universidade Federal do Rio de Janeiro 
Rio de Janeiro, Brazil 



This paper presents the results of the first part of a study 
which purpose was to reorganize and validate the Standards for Evaluations 
of Educational Programs, Projects and Materials (Joint Committee, 1981) 
for the Brazilian reality. These Standards are the result of a pioneer 
project that brought together a considerable number of American 
specialists and organizations involved in improving educational 
.evaluation activities. The Standards were originally grouped into four 
categories: utility, feasibility, propriety, and accuracy. The first one 
contains eight standards that intend to ensure that an evaluation will 
provide useful information to involved audiences. The second category 
comprises" three standards that deal with realistic, prudent, diplomatic, 
and frugal features of an evaluation^ The third category includes eight 
standards which emphasyze ethical and professional aspects of evaluation 
activities. The fourth category contains eleven standards- that call for ■ 
producing and communicating technically adequate information. For their 
importance to the evaluation area, the Standards called for further 
studdes, such as this one, which, in addition to its main purpose, 
disseminated them to a foreign public that, otherwise, would not have 
easy access to the work of the Joint Committee. 

The starting point of this study was the Joint Committee's 
recomnendation that the Standards should "be tried, reviewed and improved, 
as part of a continuing effort of many persons committed to advancing the 
practice of evaluation" (Joint Committee, 1981, p. xxi). The information 
requested by the Joint Committee included, among others, the identification 
of limitations in individual standards as well as the indication of areas 
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not covered by them. Thus, the reorganization of the Standards was based 
upon a critical review of their logical arrangement and content, and 
intends to provide data for their further refinement and revision. 

In the next secDions of this paper, the conceptual framework, 
methodological procedures, ^results, and educational importance of the 
study are reported. 

Conceptual Framework 

ft 

A preliminary examination of the evaluation standards revealed 
two principal areas of problems. The first one was related to the 
Standards classification into four categories. Due to their main 
emphasis, some standards could belong to one or another category as well. 
This problem had already been pointed out by the Joint Committee. 

The second area of problems referred to the Standards definition. 
The repetition of some aspects brought about overlapping standards. 
|Other standards presented incomplete and imprecise definitions in the 
Portuguese translation. In some cases, technical terms did not have an 
exact correspondence in English and in Portuguese; in other oases, the 
same word had different meanings in both languages. As translated into 
Portuguese, the Standards also lacked Aspects which were mentioned in the 
original version to ensure the categories intentions. 

These problems belong to the field of Logics as science. 
Science is formulated through a form of language and this language is 
assumed to be a logical one; tjius, language is conceived as logics itself 
(Cerqueira & Oliva, 1980). Consequently, the critical review of the 
Standards classification was based on logical principles such as (a) 
mutual exclusion, i.e., the elements of a division or classification must 
exclude themselves reciprocally; and (b) fundamentum divisionis which 
guides the division or classification according to the nature of the 
object being studied (Cohen & Nagel,1968; Gorsky & Tavants, 1959). It 
was assumed that these principles also guided the development of the 
original Standards. However, the need of operating exaustive and 
excluding divisions of a given object being classified according to its 
meaningful characteristics remained. 

The Standards definitions were revised in the light of a 
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linguistic perspective. According to Quine (1968), "meanings or ideas 
which are expressed through linguistic modes of identical appearance and 
enunciation may radically differ in may cases" (p. 39). So, a translated 
text may contain words 'whicH have indetermined meanings and even may 
carry out a mistaken understanding about the text focus. 

it 

Methodological Procedures 

The first part of this study — Standards reorganization — comprised 
two phases: a critical review that generated the revised version of the 
standards and the content validation of this version. Different 
procedures were mainly applied to maintain, exclude^ or include elements 
or standards during the critical review. The procedures were the follow- 
ing 1 ( a ) connection — it was the conjunction of two standards considered 
'as of the same kind; (b) complementation — it occurred when the definition 
of a standard lacked sufficient completeness; ii\ some cases, standards 
were blended; in others, new elements or elements of other standards 
were incorporated into another to achieve more objectivity, detailed 
content, and adequate scope in an attempt to express the standards over- 
view and guidelines more properly; and (c) formulation — the overall 

f ' 

analysis of the categories brought about a new standard which fulfilled 
tlje intended scope of a given attribute. In addition, standards 
descriptors were revised. This was done to make descriptors more 
adequate to the modified standards or to a category. 

After elaborating the revised standards version, its content 
validity was ascertained by three Professors of the Federal University 
of Rio de Janeiro, experts in Educational Evaluation. The purpose of the 
validation procedure was to obtain clear and adequate statements of 
evaluation standards to be utilized in Brazil, considering the incipient 
stage of development of Educational Evaluation as a professional activity 
in this country. 

Results 

The revised version of the evaluation standards comprehends 20 
standards. From these, two were maintained in their original form; two 
resulted from the connection of others; and 15 resulted from complement- 



ation. Out of 17 modified standards, ten had their descriptors revised. 
Only one standard was fdnnulated and included in the Propriety category. 
Thus, 20 standards were reorganized into the four original groups that' 
correspond to the same essential characteristics which were identified 
by the Joint Committee (1981) as "necessary and sufficient for sound 
evaluation in education 11 (p. 13). Six standards were placed in the first 
category, Utility; three standards were included* in the second category, 
Feasibility; five standards were included in the Propriety category; and 
six, in the Accuracy category, the fourth one. 

The changes that occurred in the standards are described by 
category. The inclusion of new elements in the standards had the 
purpose of avoiding misinterpretations or biased interpretations of the 
standards once they were utilized. 

In the Utility category^ the standard concerned with Audience 
Identification was complemented* by elements of the Propriety standards 
dealing with Human Interactions and Conflict of Interests; elements of 
this last one also complemented the standard Evaluator Credibility. The 
standard Information Scope and Selection incorporated elements from its 
overview and guidelines, and had its descriptor modified to Information 
Adequacy. This was done in order to make the standard more explicit for 
less trained users. The standards Valuational Interpretation, Full and 
Frank Disclosure, Described Purposes and Procedures, Justified Conclusions, 
and Objective Reporting complemented the Utility standard Report Clarity, 
reentitled Report Adequacy. The new version of this standard intended to 
assemble in a single standard, even if more complex, the elements 
considered as necessary to elaborate an appropriate report evaluation. 
Also, the standard Report Dissemination was complemented by elements of 
the Propriety standards Full and Frank Disclosure and Public's Right to 
Know, while the standard Report Timeliness was complemented by the v 
standard dealing with Evaluation Impact. 

In the second category, the Feasibility standard Political 
Viability was supplied by elements of standards Objective Reporting and 
Conflict of Interest, while standards Practical Procedures and Cost 
Effectiveness incorporated new elements and the first one had its 
descriptor changed to Procedures Viability. 



In the third category/ Propriety, the standards Formal 
Obligation and Fiscal Responsibility were more detailed and the second 
one tSbk a new descriptor — Financial Responsibility. The standard 
Rights of Human Subjects was complemented by elements of the standard 
Human Interactions, while the standard Balanced Reporting was complemented 
by the standard Full and Frank Disclosure; both standards had their 
descriptors modified to Respect to Public's Rights and Balanced Report, 
respectively. A new standard was formulated for the Propriety category — 
Public and Social Responsibility — due to the need of stating a standard, 
concerned with the social commitment of evaluators. 

In the fourth category, Accuracy, the standards Object 
Identification and Context Analysis were maintained as in the original 
form and in the s^me category. The connection of the original Accuracy 
standards Valid Measurement and Reliable Measurement resulted in the 
standard Valid and Reliable Measurement, while the connection of Analysis 
of Quantitative Information and Analysis of Qualitative Information 
originated the standard Analysis of Information. Both standards were 
kept in the Accuracy category. In these cases, the connection intended 
to avoid a dissociated use of these standards. The standards Defensible 
Information Sources and Systematic Data Control incorporated new 
expliciting elements and received modified descriptors — Selection of 
Information Sources and Data Control. 

The standards Valuational Interpretation, Evaluation Impact, 
Conflict of Intejest, Full and Frank Disclosure, Public's Right to Know, 
Human Interactions, Described Purposes and Procedures, Justified 
Conclusions, and Objective Reporting were excluded from the original 
list but had their elements incorporated into the revised standards. 
The reorganized standards are presented as follows. 

Utility Standards 

1. Audience Ident if ica.t ion . Audiences involved in or affected 
by the evaluation should be identified, so that their needs and interests 
can be addressed in the planning of evaluative studies and their personal 
involvements and interactions with persons associated with the evaluation 
can be considered and dealt with openly and honestly during the evaluation 
process. 



2 * Evaluation Credibility . The persons planning and conducting 
the evaluation should be trustworthy, politically able as well as 
competent, so that their findings achieve maximum credibility and 
acceptance from groups involved in or affected by the evaluation. 

3. Information Adequacy . Information collected should be 
carefully selected to address relevant and appropriate questions about 
the object of the evaluation and should be responsive to the needs and 
interests of audiences involved in or affected by the evaluation. 

4. Report Adequacy . The evaluation report should be consistent 
and objective, and should clearly describe the object being evaluated 

and its context, the rationale, purposes, information sources, procedures, 
limitations, findings of the evaluation, criteria for value judgements 
used for interpreting findings, and justified conclusions, in a way that 
the interested and involved audiences will readily understand what was 
d^ne, why it was done, what information was obtained, what conclusions 
were drawn, and what recommendations were made. 

5. Report Dissemination . According to related principles and 
statutes, such as those dealing with public safety and the right to 
privacy, the evaluation report should be openly and honestly disseminated 
to clients and other right-to-know audiences, so that they can assess and 
use it. V 

6 » Report Timeliness . Release of reports should be timely, so 
that audiences interested in or affected by the evaluation are encouraged 
to best use t^c 'reported information. 

Feasibility Sta^ards 

1 - Frc&edures Viability . The evaluation procedures should be 
practical, so that disruption is kept to a ninimum and should be realistic 
in relation to context, time, budget, availability of data and of 
participants without compromising, however, the evaluation results. 

2 * Political Viability . The evaluation should be planned and 
conducted with atvtec ipat ion of the different positions of involved 
groups, so that th£y can be equally represented and their cooperation may 
be obtained and possible attempts by any of these groups to curtail 
evaluation effects or to bias or misapply the results can be avoided or 
counteracted. 
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3 - Cost Effectiveness . The evaluation should be economical ' 
besides including useful information of sufficient value to justify the 
resources expended. 

Propriety Standards 

h : 

T.- Formal Obligation . Obligations and rights of the formal 

parties to an evaluation (what is to be done, how, by whom, when, where, 

and how much it costs) should be identified, agreed to and formalized in 

writing, so that these parties are obligated to adhere to all conditions 

of the agreement, or formally to renegotiate it, or even to cancel it. 

2 - Respect to Publics frights . Evaluations should be designed 
and conducted so that the welfare, dignity and individual privacy of 
involved and affected persons are ' respected and protected. 

3. Balanced Report . Evaluation reports should be complete, 
honest and fair in their disclosure of the limitations of an eva^uat ion, 
as well as in their presentation of strengths and weaknesses of the 
object under investigation, so that strengths can be built upon and 
witnesses corrected. 

' \ 4 - Financial Responsibility . The evaluation allocation and 
expenditure of resources should demonstrate -parsimony , prudence, and 
ethical responsibility besides the adoption of adequate accountability 
procedures . 

5 * Public and Social Responsibility . The evaluation planning 
and conduction should demonstrate the evaluator's honesty, integrity, 
and impartiality to avoid that information and conclusions can be 
manipulated or distorted by personal values, feelings, prejudices or 
interests of the evaluator. \ 

Accuracy Standards 

!• Object Identification . The object of the evaluation 
(program, project, material) should be *suf f iciently examined, so that the 
form(s) of the object being considered in the evaluation can b<£ clearly 
identified. 

2« Context Analysis . The context in which the program, project, 
or material exists should be examined in enough detail, so that its likely 
influences on the object can be identified. * 



3. SSlfeCtion of Information Sources . The sources of information 
should' be systematically and carefully selected and should have their * 
data-gathering controlled to ensure a sufficiently trustful and defensible 
information* 

4. Valid and Reliable Measurement , The information-gathering 
instruments and procedures measurement should be carefully chosen or 
developed and only then implemented in ways that will assure that the / 
information obtained is sufficiently valid and reliable. 

5. Data Control . The data collected and processed should be 
systematically reviewed and corrected, so that they can constitute a 
sufficiently secure basis for the interpretation of the evaluation 
results • 

6. Analysis of Information . Quantitative and qualitative 
informations should be appropriately and systematically analyzed according 
to preestablished criteria to ensure careful, logic, and safe interpret- 
ations. 

Educat iona 1 Importance 

Evaluation is seen as a means to promote better understanding 
of educational activities and to improve them. In Brazil, evaluation 
practice is recent as a professional activity. The first graduate course 
in this area was created in 1978. Until now, the few studies published 
have been merely descriptive — they lack mostly technical rigor and 
founded value judgements. Thus, the status of the evaluation area in 
Brazil has directly and negatively influenced the decision making process 
in this country. 

The standards reorganization may be considered ^gnificant as 
it may offer not only guiding principles for those who deal with 
evaluation in Brazil, but also ways of increasing the credibility of 
educational evaluation studies in this country. The reorganized standards 
may represent an additional source for the dissemination of the Joint 
Committee's whole work, not yet translated into Portuguese and, therefore, 
not accessible- to all of those who work with evaluation in Brazil. 

The authors believe that this condensed version may provoke 
immediate understanding of the standards, since it, contains standards 
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definition* now complemented and detailed in Portuguese. Additionally, 
the reorganized, version may motivate probable users to utilize the 
standards and, hopefully, contribute to the improvement of evaluation 
quality in Brazil. X 
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